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Abstract: 

We transpose an optimal control technique to the image segmentation problem. The idea is 
to consider image segmentation as a parameter estimation problem. The parameter to estimate 
is the color of the pixels of the image. We use the adaptive parameterization technique which 
builds iteratively an optimal representation of the parameter into uniform regions that form a 
partition of the domain, hence corresponding to a segmentation of the image. We minimize an 
error function during the iterations, and the partition of the image into regions is optimally driven 
by the gradient of this error. The resulting segmentation algorithm inherits desirable properties 
from its optimal control origin: soundness, robustness, and flexibility. 
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Segmentation d'image par indicateurs de raffinement 

multidimensionnels 



Resume : 

Nous transposons une technique de controle optimal a la segmentation d'image. L'idee est 
de voir la segmentation d'image comme un probleme d'estimation de parametre ou le parametre 
a estimer est la couleur des pixels de l'image. Nous utilisons la technique de parametrisation 
adaptative qui construit iterativement une representation optimalc du parametre en rgions uni- 
formes formant une partition du domaine, et correspondant ainsi a une segmentation de l'image. 
Nous minimisons une fonction d'erreur au cours des iterations, et le partitionnement de l'image en 
regions est guide de faon optimale par le gradient de cette erreur. L'algorithme de segmentation 
resultant herite des bonnes proprietes de son origine controle optimal : fondement, robustesse et 
flexibilite. 

Mots-cles : segmentation d'image, paramtrisation adaptative, problme inverse, contrle optimal 
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1 Introduction 

The segmentation of an image is usually defined as the identification of subsets of points with 
similar image properties, such as the brightness or the color (local properties), or the texture 
(global property). Image segmentation is usually subdivided into two main classes: edge detection 
approaches that follow the variations of the image properties, and region-based approaches that 
follow directly the image properties, e.g. see [2 ^ 12 ^ l2l? | |2T | |2l fl |51 Il2" l |2"1 . 

Introducing an optimization process to guide a segmentation algorithm has already been in- 
vestigated. In |14) . authors introduce an energy to minimize and develop a "region merging" 
algorithm for grey level and texture segmentation. The energy method in image segmentation is 
introduced by Geman and Geman |13) . Later a large variety of energy methods has been devel- 
oped. A formulation of the image segmentation as a piecewise modeling problem combined with an 
energy method is given in |23) . Uchiyama and Arbib |30] incorporate optimization in a merge-split, 
region-based image segmentation method; their algorithm converges to a local optimum solution 
of clustering color space based on the least sum of squares criterion. 

The connection between inverse problems and image segmentation has already been noticed, 
e.g. see [5]. Furthermore, an approach close to the one we are developing can be found in [31 j . 
where the segmentation is built through a piecewise least-squares approximation of a function 
related to image intensity. But unlike ours, the refinement of the segmentation is uniform and the 
refinements are selected from a fixed predefined set. 

The optimal adaptive segmentation algorithm presented in this paper is also based on an op- 
timization process. At each step, the refinement is optimally obtained from the gradient of the 
objective function to be minimized. The resulting regions are not geometrically constrained, as 
they are not required to be connected. Noticeably, all the steps of the algorithm are meaningful, 
since the algorithm progressively goes from the coarsest segmentation to the finest one, i.e. from a 
single monochrome region up to the complete reconstruction of the original image with one region 
per distinct color. 

This leads to some kind of segmentation on demand: since the n-th iteration provides the 
segmentation into n regions, the algorithm can segment the image into any given number of zones, 
with each zone having an optimal color with respect to the overall error from the original image. 
Moreover, since each step of the algorithm selects a single region and splits it into two parts, the 
adaptive segmentation algorithm can be used to interactively refine the segmentation of an image. 

This adaptive approach was initially developed for the inverse problem of parameter estimation: 
given a forward modeling map, the multidimensional adaptive parameterization algorithm [4 solves 
the inverse problem of estimating a multidimensional distributed parameter from measured data. 
For instance, colors are defined by their three Red/Green/Blue (RGB) components, thus color 
images can be seen as three-dimensional distributed parameters. Hence, when the parameter to 
be identified is a color image and the forward map to be inverted is the identity, this optimal 
control technique provides naturally an image segmentation algorithm that refines regions guided 
by the misfit between the original and segmented image. 

As illustrated in [3], brute force application of this algorithm to image segmentation gives 
already interesting results. But the original handling of the multidimensional aspect of the pa- 
rameter proposed there was rather naive. So this aspect is revisited, developed and enriched in 
the present paper specifically for the case of image segmentation: beside vector segmentation algo- 
rithms, where each region is associated with one color, we also introduce multiscalar segmentation 
algorithms, where each color channel has its own segmentation. When transposed back to the 
general case of nonlinear inverse problem, the new algorithms are also expected to provide more 
flexibility. 

Notice that although adaptive parameterization uses so-called cuttings that can be considered 
as defining edges, this "edge detection" aspect is just an auxiliary step to define the partitioning 
of the set of pixels into regions. Thus, we still classify our image segmentation algorithm in the 
region-based techniques. 
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The paper is organized as follows. In Section [3J we explain the similarity between parameter 
estimation and image segmentation and we illustrate the usage of adaptive parameterization to 
segment a simple image. In Section[3J we give some mathematical foundations and the description 
of the optimal adaptive vector and multiscalar segmentation algorithms. In Sectional we use the 
previous algorithms to segment a color image, and compare their relative performances. 

2 From parameter estimation to image segmentation 

2.1 Parameter estimation 

Parameter estimation is a class of inverse problems, where a simulation model has been constructed 
but some of its parameters are not precisely known. It consists in trying to recover these parameters 
from a collection of input-output experimental data. 

A classical approach to this inverse problem is to formulate it as the minimization of an 
objective function defined as a least-squares misfit between experimental output data and the 
corresponding model-generated quantities: 

(1) minimize J(c) = —\\d — J-(c)\\ 2 , 

where c denotes the unknown parameter, d the data and T is an operator corresponding to the 
model describing the cause-to-effect link from c to d. 

For a large class of inverse problems, the unknown parameters are space dependent coefficients 
in a system of partial differential equations modeling a physical problem. Due to the difficulty 
and high cost of experimental measurements, the available data are often too scarce to allow for 
the determination of the parameter value in each cell of the computational grid. 

Therefore, one has to reduce the number of unknown parameters: a convenient way is to search 
for unknown distributed parameters as piecewise constant functions, as an approximation to the 
necessarily more complex but unattainable reality. 

Then parameter estimation amounts to identify both the spatial distribution of regions of 
constant value and the associated parameter values. The result of the parameter estimation is 
then a partition of the spatial domain into regions together with one (possibly multi- dimensional) 
parameter value per region. 

2.2 Image segmentation 

Let us now consider the case where the data d is an image, made of a set of pixels (picture 
elements). Each pixel is the association of a cell Xi with its corresponding color value dc 

(2) d = {{xi, di), i £ /}, where I is the set indexing the image pixels. 
We shall denote by D the domain of the image, i.e. the set of all its cells Xi m . 

(3) D = {x h iel}. 

Vector segmentation. A vector segmentation of d consists in replacing it by an image c = 
{(xi, Ci), i<El} (where c stands for "computed") with the same domain such that: 

• c has a uniform color sr (where s stands for "segmented" ) on each region R of a partition V 
of its domain D: 

(4) Ci — sr as soon as Xi € R, 

• the image c bears some resemblance to the original image d. 
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Image (vector) segmentation methods are categorized into three main classes: pixel based, edge 
based and region based approaches [TTJ [STJ . Edge detection is mainly used for gray level image 
segmentation, it consists in locating points with abrupt changes in gray level, in [TTJ authors give 
a survey on techniques for edge detection. Region based approaches include region growing, region 
splitting, region merging and their combination. They are also divided into two classes: top-down 
and bottom-up [7] . A combination of region growing and region merging techniques is presented 
in [33] . In [TBI HFJ j authors use a homogeneous criterion based on a one-dimensional histogram to 
develop a region splitting technique. 

Here we have chosen to ensure the resemblance between the segmented and original images c 
and d by requiring that their least-squares misfit is made "small" , or, in more mathematical terms: 

(5) the segmented image c minimizes J(c) = — \\di — c;|| 2 

iei 

under the constraint that the number of regions of V is "small compared to the number of pixels 
of rf" . 

Without the constraint on the number of regions, a trivial exact solution to problem ([5]) is 
the segmentation made of one region for each pixel, with the region color being the color of its 
unique pixel. However, this trivial exact solution is not satisfactory, since it is not adequate to 
the segmentation idea. It corresponds to the ultimate level of over- segmentation. 

This approach can be seen as a top-down region based one since we keep refining the regions: 
at each iteration, the algorithm detects a discontinuity in one region color sr. which give a smaller 
value to J(c), so the algorithm can also be seen as defining an implicit edge detection. 

The result of a vector segmentation of an image d is hence a partition V of its domain D into 
regions, together with the data of one color value Sj per region. In such a segmentation, all color 
components use the same partition. 

Multiscalar segmentation. Another way to obtain an image with piecewise constant colors is 
to find a different partition for each color component. We have chosen to use the RGB basis, but 
any other representation could be used as well. So we define a multiscalar segmentation of d as 
the operation which replaces d by an image c = {(xj, Cj), i£l} such that: 

• for each k = R,G, B, the component c k of c has a uniform intensity s^ on each region R of 
a partition V k of the image domain D: 

(6) for k — R,G, B one has: c\ = as soon as Xi € R, 

• the image c bears some resemblance to the original image d, 
in the sense that c solves the same optimization problem: 

(7) the segmented image c minimizes J(c) = J k ( c ) 

k=R,G,B 

where J k (c) — ^J2iei(^i ~ c i) 2 > under the same constraint that the number of regions of the 
partitions V R , V G and V B is "small compared to the number of pixels of d" . 

The approach proposed here can produce both vector and multiscalar segmentations, and 
controls the segmentation level. 

2.3 Image segmentation considered as a particular parameter estima- 
tion problem 

Similarities between the two previous sections show that vector segmentation of images can be 
seen as a parameter estimation problem, where the forward model T reduces to the identity map 
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(compare ([T]) and (O) and complete data are available (since the exact color is known for each 
cell). 

The parameter estimation problem, because of the nonlinearity of the governing model T 
and the scarcity of data, is often ill-posed and difficult to solve. In contrast, one can expect 
that the parameter estimation algorithms will work efficiently for vector segmentation, because of 
the good properties of the associated forward map T and data. So we adapt in this paper the 
multidimensional refinement indicators algorithm [4 to the specificities of vector segmentation, 
and extend it to the determination of multiscalar segmentations. 

Remark 1 The segmentation of an image can also be attempted according to other properties than 
color, for example texture. The approach developed in this paper can be adapted to these situations 
by a proper choice of the objective function 0) or 

2.4 Illustration on an example 

In the following, we apply the adaptive parameterization algorithm of [3] to perform vector seg- 
mentation of simple images. 





Figure 1: A simple example: (a) is the image to segment, and (b), (c), (d), and (e) are the four 
successive iterations of the adaptive parameterization algorithm. 



In the first example, Figure HJa) is the original image d, i.e. the data of the parameter identifi- 
cation problem. It is made of three geometrical patterns with color green, yellow and blue on top 
of a pink background. During the iterations illustrated from Figure [TJb) to Figure (He), we build 
the segmentation in an iterative way. We start with one (homogeneous) region in Figure HJb), it 
is split into two regions in Figure QTc), a new region is detected in Figure Did), an d in Figure [TJe) 
the algorithm has converged to a segmentation made of four regions, in each region the vector 
color is constant. 

This test case exemplifies the relevance of all the iterations. All iterations are meaningful: 
the first step gives a monochrome image with the optimal color (the mean color of the picture 
elements); the second step gives the best splitting of the image into two zones with the best two 
colors; and so on. 

In the second example, the image to be segmented Figure^a) is a perturbation of the previous 
image Figure [TJa) with small square inclusions in the yellow, blue, and green patterns. Iteration 
results are illustrated from Figure [U[b) to Figure [U[h) • The four first iterations do not detect the 
perturbations, and the fourth Figure [2Ie) is exactly the final segmentation of the non perturbed 
image (to be correct, only the partition associated with it is the same). The last three iterations 
successively detect the three perturbations. 
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(h) 

Figure 2: A perturbation of the simple example: (a) is the image to segment, and (b), (c), (d), (e) 
(f), (g), and (h) are the seven successive iterations of the adaptive parameterization algorithm. 



This second example shows that the larger the contrast is, the faster it is detected, which 
means that a stopping criterion could be the number of regions supposed to be the more relevant. 
Also notice that the data image is retrieved in only seven iterations, which is exactly the number 
of distinct colors in the perturbed data image. 



3 Optimal adaptive segmentation 

Adaptive segmentation algorithms have been proposed by many authors. In [TJ level sets are used 
to design a robust evolution model based on adaptation of parameters with respect to data. In 25 
the adaptivity is related to the fact that the segmentation is adapted to different regions: regions 
are segmented using an appropriate method according to their types. The adaptivity is achieved 
through the variation of the size of the neighborhoods used for the estimation of the intensity 
functions in ITS]. Finally, adaptive segmentation in [TU] is related to the use of an adaptive 
clustering algorithm [3T] to obtain the specially adaptive dominant colors. 

The starting point for the method proposed in this paper is the adaptive multidimensional 
parameter estimation algorithm proposed in [4] for the general nonlinear inverse problem of sec- 
tion It proceeds iteratively, using multidimensional refinement indicators to split at each step 
one of the regions where the multidimensional parameter is constant. This approach allows to 
control the issue of over-parameterization by stopping the refinement process when the objective 
function is of the magnitude of the uncertainty or noise level. 

As indicated in section 12.31 we shall adapt this algorithm to the specificities of image segmen- 
tation. The result appears as an iterative region-based vector segmentation method, initialized by 
a one-region segmented image of uniform color. At each iteration, it adapts the segmentation by 
splitting one region. The region to be split and the location of the splitting, taken from a set of 
user-defined admissible cuttings, are chosen according to values of exact or first order refinement 
indicators as defined in j£l |4]. These indicators are computed from the objective function ([5]) 
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or ([7]) or its gradients with respect to color components. Once the regions are given, the color 
in each region is computed by a straightforward minimization process. With this choice, the new 
segmented image is likely to produce a significant decrease of the misfit function. The algorithm is 
stopped when the desired number of regions is attained, i.e. when the segmentation is fine enough 
to retain the information of interest, and coarse enough to avoid over-segmentation. 

Note that regions need not to be necessarily connected in our approach. This feature may be 
considered a major drawback for some image segmentation applications that requires connected 
regions only. Extending our algorithm to deal with this aspect is an interesting research problem 
that has still to be investigated. 

We describe in the next sections the specification of the adaptive parameterization technique 
to the vector segmentation problems (J5]), and its extension to the multiscalar segmentation prob- 
lem ([7]). We recall that the original image is denoted by d = {{xi, d{), i£l} (data), the segmented 
image by c = {(xj, Cj), i£l} (computed). 



3.1 Case of vector segmentation 

Let V be the current partition. The associated segmented image c, solution of the least-squares 
problem §5§ for the partition V, is given by c = MpMv d, where M-p : {sr, R G P}m>c defined 
by (@| denotes the mapping that paints the pixels of each region R in V with a given color sr, 
and M-p = (MTpM-p)" 1 MTp is its least-squares pseudoinverse, which computes the mean value 
in each region. Hence once the partition V has been chosen, the best segmented image c is simply 
obtained by replacing on each region R the image color by its mean value. 

So we concentrate now on the determination of a partition V with a controlled number of 
regions and such that J(c) is as small as possible. 



3.1.1 Refinement indicators: how good is a cutting? 

Let us first denote by C the cutting which splits a region R of V into two subregions R + and R_ . 
In order to evaluate the effect of this additional degree of freedom on the decrease of the objective 
function J(c) in the vector segmentation problem ((5]), one can use refinement indicators iQ, pp. 110— 
111]: 

Exact indicators. They consist in 

(8) AJ = J(c) - J(c c ), 

where cc is the solution of problem ([5]) for the partition Vc obtained by splitting region R 
of V into R + and R . These indicators are called "nonlinear" in [5] because in the general 
case where J- is nonlinear, the evaluation of cc requires the resolution of the full nonlinear 
least-squares problem ((T|) with the tentative refined partition Vc- This prohibits their use 
for the test of a large number of cuttings. 

On the contrary, for our problem where T reduces to the identity map, J is additive with 
respect to regions of V, so that cc coincides with c outside of R. Its value on R is given by 
the simple formulas: 

, . i \ ) c + f° r x i ^ where c+ = mean value of {dj,xj G R+}, 

^ ' \ c C)i \ c _ f or x . g where c_ = mean value of {dj,Xj G R_}. 

Hence A J can be computed at a very low cost. This makes it possible to use exact indicators 
to rank a quite large number of tentative cuttings. 

First order indicators, as defined in [3] for multidimensional parameters, are given by: 

(10) ||A|| 9) l<q<™, X = (X R , A G , X B ), 

where the q is user defined and 

f) jk Q jk 

(ID * k = E %T~ E f° r k = R,G, B. 
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Because of the simple form of J k , the following relation holds for each color component 
(see g], or [SJ p. 126] for the proof): 

(12) A J fc = ^^ for k = R,G,B, 

8 p\p k _ 

where p is the number of points Xi in R and pH and p_ are the numbers of points x\ in the 
tentative subregions R + and R . 
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Figure 3: Comparison of exact and first order indicators for various p. Each curve corresponds to 
the fy-^fPr function defined in (TH 



In order to illustrate the quality of this first order indicator, we have plotted in Figure [3] the 
curved 

(13) ee [ ,i]^p r |e< i^/AJ fe <l| 

for regions R with different numbers p of pixel, under the hypothesis that all partitions of R 
are equiprobable. As one can see, this probability is practically one for £ = 0.9 as soon as R 
has more than 80 pixels. This shows that the first order indicators |A fc | are well suited to 
rank cuttings inside a given region R. It also shows that they are not suited to rank cuttings 
belonging to different regions, unless they are replaced by \X k \/p^ , where p is the number of 
pixel of the region subject to the cutting. 

The computational cost of first order indicator is here similar to that of exact indicators, so 
one can wonder why to use an approximation when an exact calculation is available at the same 
price. The answer will appear in the next section. 

3.1.2 Cutting strategies 

Now that we know how to rank cuttings, the next step is to specify the cutting strategy, i.e. the 
family T of cuttings C to be tested for the splitting of each region. 

1 Since in I12I I. we have p = p* + p* , we also have P ^>~ = ( 1 — ^j- J < i, and thus A J k > . 
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For nonlinear problems, adaptive parameterization has been shown to be regularizing when 
the lamily T is adapted to the problem [5], But if enriching the cutting family always provides 
a better local decrease of the cost function, it can also decrease the regularizing effect by leading 
the algorithm into a local minimum. 

However, for our segmentation problem, the forward map T is the identity, so there is no 
danger of local minima, and we are free to choose the family T at our convenience, even if it is 
extremely large. So we shall consider two cutting strategies: 

Best in a family. The set T of cuttings used for the tentative splitting of a region is a user- 
defined family of cuttings. It may incorporate any a priori information on the parameter 
when such information is available. 

We have used in our numerical results the set T made of all vertical and horizontal cut- 
tings, which split any region R in two subregions separated by one vertical or horizontal 
boundary. This family of cuttings has proven useful for the nonlinear parameter estimation 
problem, where one seeks to describe the parameter by regions with simple forms (see for 
example [5], where T contained also diagonal and checkerboard cuttings). But we shall see 
in the numerical results section that it seems much less adapted to image segmentation. 

In any case, the number of cuttings to be tested with this strategy is bounded from above by 
the sum of the number of horizontal and vertical pixels in the image to be segmented. Hence 
the best tentative cutting Cj^ of region R can be determined by an exhaustive computation 
of all exact indicators A J associated with all cuttings C of R in the chosen family T, as they 
are not more expensive as the first order ones: 



(14) 



Cj^ maximizes A J over all cuttings CsT of R. 



Overall best. Here T is made of all possible 2-partitions of the region R. So the number of 
cuttings to be tested for the determination of Cj^ is 2 card ( R ), which becomes extremely large 
as soon as R contains enough points. Hence one cannot afford to perform an exhaustive 
computation of all corresponding indicators, let them be exact or first order. So this cutting 
strategy can be used only in cases where an analytical determination of the cut Cj^ which 
maximizes AJ or ||A|| g over T is available. To our best knowledge, such analytical solution 
is available neither for the exact indicator AJ, nor for the refinement indicators ||A|| g for 
any q, except for q = oo. This is where the first order indicators play a crucial role in image 
segmentation. 

We describe now the analytical determination of Cj^ when q = oo. As it was noticed in [4], 
formula (TTTj) shows that, for each color component k, the cutting which maximizes |ApJ 
is obtained by splitting R according to the sign of dJ k /dc k : 



(15) 



c*4 = 



eR 



>dJ k 
dc k 



> 



eR 



«dJ k 



< 



The associated first order indicators of largest modulus for the splitting of R are then, for 
each component: 



(16) 



A R - 



A R I - 



xiER 



dJ k 



dc k 



for k = R,G, B. 



So if we denote by k* the color component with the largest |A^ fc |, one has: 



(17) 



|A|| 
|A|| 



del 



max 

k=R,G,B 



A 



R 



< max 

k=R,G,B 



V R 



R 



for the cutting C = C, 



*,k* 
R 



This shows that 



def 



/ R maximizes 1 1 A 1 1 ^ over the set T of all 2-partitions of the region R. 

Since we have no theoretically funded way to choose a predefined set of cuttings among which 
to select the refinement, the overall best cutting strategy will be the default option for image 
segmentation. 
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3.1.3 Updating the partition V 

The algorithm computes the optimal cuttings Cj^ for each region R of the current partition V 
as indicated in the previous section 13.1.21 and determines the region R* with the largest exact 
refinement indicator. 

The next partition V* is then obtained by splitting region R* according to cutting Cj^* , thus 
increasing the number of regions by one. The new segmented image c* is updated by formula (O 
over the region R*, and coincides with c over all other regions. 

After n iterations, the vector segmentation algorithm adds n regions to the initial partition. 

3.1.4 Optimal adaptive vector segmentation algorithm 

We summarize here the algorithm for the vector segmentation of an image d. Let 1(C) denote 
the best available refinement indicators for the ranking of cuttings C in the chosen family T (see 
section I3.1.2j) . 

Specify the cutting strategy. Choose one from: 

• best in a family (and then 1(C) — A J = exact indicators), 

• overall best (and then 1(C) = HAHoo = first order indicators). 

Initialization. 

-1. Define the initial partition Vi as the 1-region partition of the domain D of d. 

0. Compute the associated 1-region segmented image: 

ci = Mv x M-Pi d Vz 6 /, (c\)i = mean value of {dj,j E I}. 

Iterations. For n>l, do until the desired number of regions is attained: 

1. For all regions R of the current partition V n , 

la. determine the best cutting Cj^ using the best available indicator 1(C), 
lb. compute the exact indicators A J associated with Cj^ using ([5]) and 

2. Retain the region R* whose best cutting Cj^, has the largest exact indicator A J. 

3. Define the next partition V n +i by splitting R* according to cutting C^. 

4. Define the next segmented image c n +i — Mv n+1 -Mv n+ i d by updating c„ on R* ac- 
cording to the tentative segmented image on R* obtained during step lb. 

Remark 2 When T is not the identity operator, one can use, for the best in a family cutting 
strategy, the first order indicators in step la instead of the exact ones if the cost of an exhaustive 
search with the exact indicators is too high (see the adaptive parameterization algorithm UV). 

Remark 3 It is of course possible to start the algorithm from any partition V\ instead of a 1- 
partition. 

3.2 Case of multiscalar segmentation 

In order to construct a multiscalar segmentation, one notice that (|7|) is equivalent to 

(18) minimize J k (c k ) = i ^(d k - c k f for k = R,G, B, 

iei 

under the same constraint that the number of regions of the partitions V is "small compared to 
the number of pixels of d" . 

This suggests to apply a scalar version of the previous optimal adaptive segmentation algo- 
rithm independently to each component k = R,G, B. If V = (V R ,V ,V B ) denotes the current 
multiscalar partition, this will produce three tentative optimal partitions (V R ^, V G ^ , V B ^). It 
will then be necessary to choose one multiscalar strategy to decide how to use this information to 
define the updated multiscalar segmentation V* = (V R * ,V G * ,V B *). 
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3.2.1 Scalar optimal adaptive segmentation 

For a given color component fc, the scalar algorithm follows the steps of section 13.1.41 with the 
following minor adaptations. 

Let V k be the current partition for component fc. The associated fc-th component seg- 
mented image, solution of (|18|) for the partition V k , is now given by c k = MpkMpkd k , where 
M-pk : (s^,R £ V k )*-tc k denotes the mapping that attributes the segmented intensity to 
color component k on each region R of partition V k , and M-pk — ((Mpk) T Mpk) (Mpk) T 
its least-squares pseudoinverse, which computes the mean value of the fc-th component over each 
region. 

The refinement indicators associated with a cutting C which splits a region R of the current 
fc-th component partition V k into two subregions R+ and R_ are (compare with section 13.1.11) : 

Exact indicators: 

(19) AJ k = J k (c k ) - J k (c k c ), 

where c k is the fc-th color component of the current segmented image c, and c k c minimizes J k 
defined in (fT5)) for the partition Vq obtained by splitting region R of V k into R+ and R . 
As in the vector case, is easily computed on R by: 



c 



C 

j c+ for Xi £ R+ where = mean value of {d k ,Xj G R+}. 
1 ' ' I r k for Xi £ R_ where c k _ = mean value of {dh, Xj G R_}. 



First order indicators: 

(21) |A fe | with A fc given by (fTTI) . 

Of course, exact and first order indicators are still linked by relation (fT2"j). 

In the absence of specific information for each component, we have chosen to use the same 
set T of tentative cuttings for each color component k. So the cutting strategies are (compare to 
section I3.1.2j) : 

Best in a family. T is a predefined family of cuttings, for example the set of vertical and hori- 
zontal cuttings. With this strategy, the best cutting Cj^ of a region R of V k for the exact 
indicator AJ k needs to be computed by an exhaustive search. 

Overall best. T is made of all possible 2-partitions of the region R. Here the best cutting Cj^ 
for the first order indicator s\\ k \ can be determined analytically by formula (1161) . 

So after one iteration of the scalar algorithm has been performed on each component fc, the 
following quantities are available: 

(22) V k ^ = tentative optimal partition for color component fc, 

(23) AJ k # = exact decrease of the minimum of J k when V k is replaced by V k # . 

3.2.2 Updating the partition V ~ (V R ,V G ,V B ): multiscalar strategy 

We describe now three multiscalar strategies which can be applied to update the current multiscalar 
partition V once the results (|2"21 and ([2"B"f of the scalar algorithm have been obtained for each color 
component: 

Best component only. This strategy determines the color component fc* whose tentative re- 
finement produces the largest decrease AJ k #, and applies the corresponding optimal parti- 
tion V k # to component fc* only. When fc* = R (red) for example, the current and updated 
multiscalar partitions are: 

(24) V — (V R , V G ,V B ), V* = (V R * ,V G ,V B ). 
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After n iterations, the best component only multiscalar strategy adds n regions to the initial 
multiscalar partition. Of course, the vector partition corresponding to the resulting multi- 
scalar partition, obtained by superimposing the three scalar partitions, may contain many 
more regions. 

Best component for each. In this strategy, the algorithm refines each component k according 
to the corresponding tentative optimal partition "P k ^ determined by the scalar algorithms. 
The current and updated multiscalar partitions are then: 

(25) V = (v R ,r G ,r B ), V* = (V R *,V G *,V B *). 

After n iterations, the best component for each multiscalar strategy has added 3n regions to 
the initial partition (n regions per component). Again, the vector partition corresponding 
to the resulting multiscalar partition, obtained by superimposing the three scalar partitions, 
may contain many more regions. 

The best component for each multiscalar strategy amounts to apply the scalar refinement 
indicator algorithm to each component separately, i.e. to use the tensor product of the three 
algorithms. 

Combine best components. Here the algorithm selects the three optimal cuttings Cj£t corre- 
sponding to the best tentative segmentation for each component k = R,G, B, and applies all 
of them to all components. This makes sense only if the three current component partitions 
coincide, and produces a partition f> RGB # i which is nothing but the superimposition of the 
three tentative partitions J> R # , V G ^ and V B #. This partition is used to update all three 
components: 

(26) V = (V R ,P G ,V B ), V* = (V RGB * ,V RGB * ,T> RGB #). 

So the combine best components multiscalar strategy constructs in fact a sequence of vector 
segmentations of the original image d. 

In contrast with the vector segmentation algorithm of section l3.1.4l the combine best compo- 
nents multiscalar strategy adds at each iteration between 3 and 2 3 — 1 regions to the current 
vector partition. So the number of regions may increase rapidly, which is not compatible 
with the idea of adaptive parameterization. 

Note that all multiscalar strategies are equivalent in the scalar case. 
3.2.3 The multiscalar adaptive segmentation algorithm 

We summarize here the adaptation of the algorithm of section 13.1.41 to the determination of 
multiscalar segmentations. 

Specify the cutting strategy. Choose one from: 

• best in a family (best available = exact indicators I k (C) = AJ fc ), 

• overall best (best available = first order indicators I k (C) = \X k \). 

Specify the multiscalar strategy. Choose one from: 

• best component only, 

• best component for each, 

• combine best components. 

Initialization. Define Ti = {V R ,V G ,Vf) and c a = (cf,cf ,cf ) by: 

Scalar initializations. For each component k ~ R,G, B: 

-1. Define the V k as the 1-region partition of the domain D of d. 
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0. Compute the associated 1-region segmented component image: 

c\ = M V k M V k d k Mi £ I, (c\)i = mean value of {d k , j e I}. 
Iterations For n>l, do until the desired number of regions is attained: 

Scalar iterations. For each component k = R,G,B, do (section 13. 2. 

1 . For all regions R of the current partition V k , 

la. determine the best cutting Cj^ using the best available indicator 1(C), 

lb. compute the exact indicators AJ k associated with using (fl"9|) and ((20|) . 

2. Retain the region R* whose best cutting has the largest exact indicator AJ k . 

3. Define the next tentative partition V k # of color component k by splitting the 
region R* of V k according to cutting Cj£ . 

Update the multiscalar segmentation. 

4. Compute the next multiscalar partition V n +i according to the chosen multiscalar 
strategy (section f3. 2. 2 1) . 

5. Define the next multiscalar segmented image c n +i: 

• best component only: let k* be the component retained at step 4. It suffices 
then to update c\ on the region R* determined at step 2 according to the 
tentative segmented image (|2"0)) determined at step lb. 

• best component for each: each component k of c\ needs to be updated on 
the corresponding region R* determined at step 2 according to the tentative 
segmented image (|2T))) obtained during step lb. 

• combine best components: here c n +i is obtained by updating c n on the 1 to 3 
regions R E V n which have been split to define V n +i- 

Remark 4 It is of course possible to start the algorithm from any multiscalar partition V\ instead 
of three 1-partitions, with the restriction that V\ has to be a vector partition in case the combine 
best components multiscalar strategy is used. 

Remark 5 Note that for both the vector and multiscalar segmentations, the expressions to com- 
pute are simple closed-form formulas that are local to regions. Furthermore, in steps la and lb the 
best cutting for each region and its associated exact indicator are invariant as long as the region 
is not split. Hence, an iteration of the algorithm only amounts to compute step 1 for the two new 
subregions ( of the retained region R* ) that appeared in the partition V n ot the previous iteration. 
Taking into account this optimization, it is obvious to derive a linear version of the algorithm (in 
terms of the number of iterations) , which is a salient aspect of this approach. 

4 Numerical results 

We illustrate in this section the numerical behavior of algorithms of sections 13.1.41 and 13.2.31 for 
the adaptive vector and multiscalar segmentations. The image d to be segmented is the picture of 
a colorful sculpture [TS]. Its definition is 400 x 533 (i.e., 213,200 pixels). 

Note that the original image d is itself trivially a segmented image, with a number of regions 
equal, for a vector segmentation, to the number of distinct colors — here 55,505 — , or equal, for each 
component for a multiscalar segmentation, to the number of distinct levels in the component — 
here the maximum of 256 levels for a 24-bit color image is reached for all components. Hence, 
if no constraint is set on the number of regions, all above adaptive segmentation algorithms will 
converge towards the original image d. But, though regions are introduced in our algorithms with 
parsimony at each iteration, there is for the time no mathematical proof that they will produce a 
segmented image c identical to d as soon as the number of regions in c becomes equal to that of 
the trivial segmentation of d. But c will for sure end up being identical to d at some point before 
the number of its regions equals the number of its pixels. 
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This convergence property is important as it ensures that by properly stopping the algorithm, 
one can control the compromise between the number of regions of the segmentation and the 
resemblance of the segmented image to the original one. 

The software used to compute these numerical examples is still under development, but it will 
be available soon at http://refinement.inria.fr/. 

4.1 Vector segmentation 

We test first the vector segmentation algorithm 13.1.41 with two cutting strategies. Here the al- 
gorithm adds exactly one region at each iteration, so it is easy to specify a stopping criterion 
providing a segmentation into exactly n regions with uniform colors. 

4.1.1 Best in a family cutting strategy 




Figure 4: Vector segmentation with the best in a family cutting strategy, (a) is the image data 
to segment. Then, for each displayed image, we give the iteration number n, the number of 
regions n vr , and the percentage of explained data r. 

(b) n = 0, n vr = 1, t = 65.0%; 

(c) n= 1, n vr =2, r = 65.1%; 

(d) n = 10, n vr = 11, t = 68.7%; 

(e) n = 40, n vr = 41, r = 73.4%; 

(f) n = 200, n„ = 201, r = 82.2%. 

The images shown in Figure [4] are obtained with a quite scarce predefined family of vertical 
and horizontal cuttings: regions are rectangles, and the only cuttings to be tested in each region 
are its horizontal and vertical divisions into two equal parts. For instance, the first cutting is 
vertical as shown in Figure 01c), and all regions in Figures IDJd), HJe) and 0Jf ) are rectangular. 
The exact indicators are used to rank all cuts. 

The adaptive character of the algorithm is clear on images Figures Efd), EJe) and[DJf) where 
regions are extremely irregular. At each iteration, the cutting providing the largest decrease of 
the cost function is selected. 

The convergence of the algorithm is very slow: the image Figure 2{f) obtained after 200 it- 
erations is made of 200 monochrome rectangles, but it bears only a rough resemblance to the 
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data. So it seems that the best in a family cutting strategy is not well suited for segmentation 
purpose — even though it remains a very good strategy for the general problem of vector parameter 
estimation. 

4.1.2 Overall best cutting strategy 




Figure 5: Vector segmentation with the overall best cutting strategy, (a) is the image data to seg- 
ment. Then, for each displayed image, we give the iteration number n, the number of regions n vr , 
and the percentage of explained data r. 

(b) n = 0, n vr = 1, t = 65.0%; 

(c) n= 1, n vr = 2, t = 77.9%; 

(d) n = 2, n vr = 3, r = 82.1%; 
(c) n = 5, n vr = 6, r = 86.8%; 

(f) n = 7, n vr = 8, r = 88.1%; 

(g) n = 10, n vr = 11, t = 89.6%; 

(h) n = 40, n vr = 41, t = 94.8%. 

We present in Figure [5] the vector segmentation of the same image with the overall best cutting 
strategy. Here the shape of the regions are perfectly adapted to the image data: the image is 
already recognizable with only three regions, see Figure EJd), and the algorithm is able to explain 
86.8% of the data with as few as six color regions, see Figure [5je). But it takes 41 color regions to 
explain up to 94.8% of the data. In any case, the intermediate images generated by the algorithm 
offer a large choice of segmented images to choose from. 

It is this configuration of the algorithm (vector segmentation with overall best cutting strategy) 
which was used to run the example of Figure [21 There also, as we have seen in section 12.41 the 
intermediate iterations are pertinent as they let us discover progressively the smaller perturbations. 

In conclusion, the overall best cutting strategy seems to be well adapted to image segmentation 
purposes, so we have used it in all the numerical results below. 

4.2 Multiscalar segmentation 

For the general parameter estimation problem, multiscalar segmentation is a natural approach 
when each component of the vector parameter has a different physical interpretation. This is not 
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the case for image segmentation, where the three color components have the same status. But 
one can expect that allowing a different partition for each component will allow to represent the 
image with a smaller number of regions for a same level of details. All tests in this section have 
been performed with the overall best cutting strategy. 

4.2.1 The best component only multiscalar strategy 




Figure 6: Multiscalar segmentation with the overall best cutting strategy and the best component 
only multiscalar strategy, (a) is the image data to segment. Then, for each displayed image, we 
give the iteration number n, the number of scalar regions n sr , the resulting number of uniform 
color regions n vr , and the percentage of explained data r. 



(b) 


n 


= 0, n sr 


= 3, 


n vr 


= 1,T = 


65.0%; 


(c) 


n 


= 1, n sr 


= 4, 


7Z V r 


= 2,T = 


70.0%; 


(d) 


n 


— 2, 71 sr 


= 5, 


n V r 


= 4, T = 


75.6%; 


(e) 


n 


— 3, n SY 


= 6, 




= 8,T = 


82.8%; 


(f) 


n 


= 6, n sr 


= 9, 


Ti vr 


= 26, t = 


= 90.1% 



Here one iteration of the algorithm adds exactly one scalar degree of freedom to the component 
which produces the largest decrease of the objective function. Hence, one can expect to obtain 
a good representation of the original image with fewer component regions. This is confirmed by 
the comparison of Figures [6jf) and[5]Jg): the best component only multiscalar strategy explains 
90% of the data with only 9 scalar regions (which generate 26 color regions), whereas the vector 
segmentation algorithm requires 11 color regions, which correspond to 33 component regions. 

4.2.2 The best component for each multiscalar strategy 

The control on the number of degrees of freedom is less precise with this algorithm, as one iteration 
adds exactly three scalar degrees of freedom instead of one for the previous algorithm. 

Note that the images of Figures [7](d) and[6jf) coincide: this means simply that the three first 
iterations — after the three initialization ones — of the best component only algorithm have updated 
successively the R, G and B components. But when the number of iterations increases, it can be 
expected that the best component only algorithm will not cycle regularly through the R, G and B 
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Figure 7: Multiscalar segmentation with the overall best cutting strategy and the best component 
for each multiscalar strategy, (a) is the image data to segment. Then, for each displayed image, 
we give the iteration number n, the number of scalar regions n sr , the resulting number of vector 
regions n vr , and the ratio of explained data r. 

(b) n = 0, n SI = 3, n VI = 1, r = 65.0%; 

(c) n = 1, n SI = 6, n„ = 8, r = 82.8%; 

(d) n = 2, n SI = 9, n vr = 26, r = 90.1%; 
(c) n = 5, n SI = 18, n vr = 153, r = 94.7%; 
(f) n = 10, n sr = 33, ra vr = 4581, r = 97.1%. 



components as the best component for each does, and hence produce better images for a given 
number of component regions. 

Because of the complete independence of the partitions for the R, G and B components, 
the number of color regions obtained by superimposition of the component regions blows up for 
these two algorithms, see Figure E£f). This shows that the parameterization of the image by 
component regions instead of color regions allows a drastic reduction of the number of regions for 
a given image quality. In other words, it is possible with these multiscalar algorithms to obtain 
a segmented image with a large number of color regions, described by a much smaller number of 
component regions. 

4.2.3 The combine best components multiscalar strategy 

This multiscalar strategy produces in fact a vector segmentation, as we have seen in the last item 
of section f3. 2. 21 Hence, all current component images are segmented on the same partition, so we 
have omitted in Figure [5] the number n sr of scalar regions, as it is given by n sr = 3n vr - 

Comparison of Figures [5Jd) and[5Je) with Figure [^g) shows that, for a given level of accuracy, 
the combine best components multiscalar strategy tends to produce vector segmented images with 
a larger number of regions than the direct vector segmentation algorithm. 

Also, the algorithm adds at each iteration a number of color regions between 3 and 7, i.e. 
between 9 and 21 scalar degrees of freedom, which makes it more difficult to control of the number 
of color regions. 

So in a whole, the combine best components multiscalar strategy seems less performing for 
image vector segmentation than the direct vector segmentation algorithm l3.1.4l 
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Figure 8: Multiscalar segmentation with the overall best cutting strategy and the combine best 
components multiscalar strategy, (a) is the image data to segment. Then, for each displayed 
image, we give the iteration number n, the number of uniform color regions n vr , and the percentage 
of explained data r. 



(b) 


n = 


0, n VI = 


l,r = 


65.0%; 


(c) 


n = 


1, n vr = 


8, T = 


84.1%; 


(d) 


n = 


2, n VI = 


12, T = 


= 88.3%; 


(e) 


n = 


5, 71 V r — 


24, t -- 


= 91.4%; 


(f) 


n = 


10, n vr - 


= 42, t 


= 94.1%. 



Conclusion 

Once recognized as a parameter estimation problem, for which the forward modeling map to invert 
is the identity, image segmentation may benefit from the theory of optimal control. 

We have studied the specificities of this favorable case in the context of adaptive parameteriza- 
tion and we have proposed an optimal adaptive (vector) segmentation algorithm. This algorithm 
provides a finite sequence of segmented images with a number of color regions increasing by one 
at each iteration. Thanks to its adaptive character, the algorithm extracts the significant regions 
from the very beginning, and all the intermediate steps of the process are meaningful. The opti- 
mal character indicates that the most significant regions are extracted first, and that the sequence 
minimizes the error with respect to the original image and terminates with a segmented image 
containing as many regions as distinct colors in the original image corresponding to the null er- 
ror. This algorithm is well-suited to finely control the desired number of color regions, since it is 
equipped with a meaningful criterion to stop the refinement process. 

The study has also exhibited a class of multiscalar algorithms that provide a distinct segmen- 
tation for each color component. Here the number of component regions increases by one or three 
at each iteration, but the number of resulting color regions blows up quickly, thus explaining a 
large percentage of the data. Hence, this algorithm also shines when a desired percentage of the 
data has to be explained with a number of (component) regions as small as possible. 
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